Predict Bike Sharing Demand with AutoGluon Template

Project: Predict Bike Sharing Demand with AutoGluon

This notebook is a template with each step that you need to complete for the project.

Please fill in your code where there are explicit ? markers in the notebook. You are welcome to add more cells and code as you see fit.

Once you have completed all the code implementations, please export your notebook as a HTML file so the reviews can view your code. Make sure you have all outputs correctly outputted.

File-> Export Notebook As... -> Export Notebook as HTML

There is a writeup to complete as well after all code implememtation is done. Please answer all questions and attach the necessary tables and charts. You can complete the writeup in either markdown or PDF.

Completing the code template and writeup template will cover all of the rubric points for this project.

The rubric contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this notebook and also discuss the results in the writeup file.

Step 1: Create an account with Kaggle

Create Kaggle Account and download API key

Below is example of steps to get the API username and key. Each student will have their own username and key.

1. Open account settings.
2. Scroll down to API and click Create New API Token.
3. Open up kaggle.json and use the username and key.

Step 2: Download the Kaggle dataset using the kaggle python library

Open up Sagemaker Studio and use starter template

1. Notebook should be using a ml.t3.medium instance (2 vCPU + 4 GiB)
2. Notebook should be using kernal: Python 3 (MXNet 1.8 Python 3.7 CPU Optimized)

Install packages

!pip install -U pip
!pip install -U setuptools wheel
!pip install -U "mxnet<2.0.0" bokeh==2.0.1
!pip install autogluon --no-cache-dir
# Without --no-cache-dir, smaller aws instances may have trouble installing

Setup Kaggle API Key

# create the .kaggle directory and an empty kaggle.json file
!mkdir -p /root/.kaggle
!touch /root/.kaggle/kaggle.json
!chmod 600 /root/.kaggle/kaggle.json

# Fill in your user name and key from creating the kaggle account and API token file
import json
kaggle_username = "name"
kaggle_key = "key"

# Save API token the kaggle.json file
with open("/root/.kaggle/kaggle.json", "w") as f:
    f.write(json.dumps({"username": kaggle_username, "key": kaggle_key}))

Download and explore dataset

Go to the bike sharing demand competition and agree to the terms

# Download the dataset, it will be in a .zip file so you'll need to unzip it as well.
!kaggle competitions download -c bike-sharing-demand
# If you already downloaded it you can use the -o command to overwrite the file
!unzip -o bike-sharing-demand.zip

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

from autogluon.tabular import TabularPredictor

# Create the train dataset in pandas by reading the csv
# Set the parsing of the datetime column so you can use some of the `dt` features in pandas later
train = pd.read_csv("data/train.csv", parse_dates=["datetime"])
train.head()

	datetime	season	holiday	workingday	weather	temp	atemp	humidity	windspeed	casual	registered	count
0	2011-01-01 00:00:00	1	0	0	1	9.84	14.395	81	0.0	3	13	16
1	2011-01-01 01:00:00	1	0	0	1	9.02	13.635	80	0.0	8	32	40
2	2011-01-01 02:00:00	1	0	0	1	9.02	13.635	80	0.0	5	27	32
3	2011-01-01 03:00:00	1	0	0	1	9.84	14.395	75	0.0	3	10	13
4	2011-01-01 04:00:00	1	0	0	1	9.84	14.395	75	0.0	0	1	1

# Simple output of the train dataset to view some of the min/max/varition of the dataset features.
train.describe()

	season	holiday	workingday	weather	temp	atemp	humidity	windspeed	casual	registered	count
count	10886.000000	10886.000000	10886.000000	10886.000000	10886.00000	10886.000000	10886.000000	10886.000000	10886.000000	10886.000000	10886.000000
mean	2.506614	0.028569	0.680875	1.418427	20.23086	23.655084	61.886460	12.799395	36.021955	155.552177	191.574132
std	1.116174	0.166599	0.466159	0.633839	7.79159	8.474601	19.245033	8.164537	49.960477	151.039033	181.144454
min	1.000000	0.000000	0.000000	1.000000	0.82000	0.760000	0.000000	0.000000	0.000000	0.000000	1.000000
25%	2.000000	0.000000	0.000000	1.000000	13.94000	16.665000	47.000000	7.001500	4.000000	36.000000	42.000000
50%	3.000000	0.000000	1.000000	1.000000	20.50000	24.240000	62.000000	12.998000	17.000000	118.000000	145.000000
75%	4.000000	0.000000	1.000000	2.000000	26.24000	31.060000	77.000000	16.997900	49.000000	222.000000	284.000000
max	4.000000	1.000000	1.000000	4.000000	41.00000	45.455000	100.000000	56.996900	367.000000	886.000000	977.000000

# Create the test pandas dataframe in pandas by reading the csv, remember to parse the datetime!
test = pd.read_csv("data/test.csv", parse_dates=["datetime"])
test.head()

	datetime	season	holiday	workingday	weather	temp	atemp	humidity	windspeed
0	2011-01-20 00:00:00	1	0	1	1	10.66	11.365	56	26.0027
1	2011-01-20 01:00:00	1	0	1	1	10.66	13.635	56	0.0000
2	2011-01-20 02:00:00	1	0	1	1	10.66	13.635	56	0.0000
3	2011-01-20 03:00:00	1	0	1	1	10.66	12.880	56	11.0014
4	2011-01-20 04:00:00	1	0	1	1	10.66	12.880	56	11.0014

# Same thing as train and test dataset
submission = pd.read_csv("data/sampleSubmission.csv", parse_dates=["datetime"])
submission.head()

	datetime	count
0	2011-01-20 00:00:00	0
1	2011-01-20 01:00:00	0
2	2011-01-20 02:00:00	0
3	2011-01-20 03:00:00	0
4	2011-01-20 04:00:00	0

Step 3: Train a model using AutoGluon’s Tabular Prediction

Requirements:

• We are prediting count, so it is the label we are setting.
• Ignore casual and registered columns as they are also not present in the test dataset.
• Use the root_mean_squared_error as the metric to use for evaluation.
• Set a time limit of 10 minutes (600 seconds).
• Use the preset best_quality to focus on creating the best model.

predictor = TabularPredictor(
    label="count", 
    problem_type="regression", 
    eval_metric="root_mean_squared_error",
    learner_kwargs={"ignored_columns": ["casual", "registered"]}
).fit(
    train_data=train,
    time_limit=600,
    presets="best_quality"
)
                                                                                          

No path specified. Models will be saved in: "AutogluonModels/ag-20220403_215705/"
Presets specified: ['best_quality']
Beginning AutoGluon training ... Time limit = 600s
AutoGluon will save models to "AutogluonModels/ag-20220403_215705/"
AutoGluon Version:  0.4.0
Python Version:     3.9.7
Operating System:   Darwin
Train Data Rows:    10886
Train Data Columns: 11
Label Column: count
Preprocessing data ...
Using Feature Generators to preprocess the data ...
Dropping user-specified ignored columns: ['casual', 'registered']
Fitting AutoMLPipelineFeatureGenerator...
	Available Memory:                    391.59 MB
	Train Data (Original)  Memory Usage: 0.78 MB (0.2% of available memory)
	Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
	Stage 1 Generators:
		Fitting AsTypeFeatureGenerator...
			Note: Converting 2 features to boolean dtype as they only contain 2 unique values.
	Stage 2 Generators:
		Fitting FillNaFeatureGenerator...
	Stage 3 Generators:
		Fitting IdentityFeatureGenerator...
		Fitting DatetimeFeatureGenerator...
	Stage 4 Generators:
		Fitting DropUniqueFeatureGenerator...
	Types of features in original data (raw dtype, special dtypes):
		('datetime', []) : 1 | ['datetime']
		('float', [])    : 3 | ['temp', 'atemp', 'windspeed']
		('int', [])      : 5 | ['season', 'holiday', 'workingday', 'weather', 'humidity']
	Types of features in processed data (raw dtype, special dtypes):
		('float', [])                : 3 | ['temp', 'atemp', 'windspeed']
		('int', [])                  : 3 | ['season', 'weather', 'humidity']
		('int', ['bool'])            : 2 | ['holiday', 'workingday']
		('int', ['datetime_as_int']) : 5 | ['datetime', 'datetime.year', 'datetime.month', 'datetime.day', 'datetime.dayofweek']
	0.1s = Fit runtime
	9 features in original data used to generate 13 features in processed data.
	Train Data (Processed) Memory Usage: 0.98 MB (0.2% of available memory)
Data preprocessing and feature engineering runtime = 0.09s ...
AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error'
	To change this, specify the eval_metric parameter of Predictor()
AutoGluon will fit 2 stack levels (L1 to L2) ...
Fitting 11 L1 models ...
Fitting model: KNeighborsUnif_BAG_L1 ... Training model for up to 399.84s of the 599.91s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
	-101.507	 = Validation score   (root_mean_squared_error)
	0.04s	 = Training   runtime
	0.03s	 = Validation runtime
Fitting model: KNeighborsDist_BAG_L1 ... Training model for up to 399.76s of the 599.83s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
	-84.1058	 = Validation score   (root_mean_squared_error)
	0.02s	 = Training   runtime
	0.03s	 = Validation runtime
Fitting model: LightGBMXT_BAG_L1 ... Training model for up to 399.7s of the 599.77s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMXT_BAG_L1 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: LightGBM_BAG_L1 ... Training model for up to 399.6s of the 599.67s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBM_BAG_L1 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: RandomForestMSE_BAG_L1 ... Training model for up to 399.59s of the 599.66s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 67 due to low memory. Expected memory usage reduced from 66.52% -> 15.0% of available memory...
	-119.0818	 = Validation score   (root_mean_squared_error)
	0.43s	 = Training   runtime
	0.06s	 = Validation runtime
Fitting model: CatBoost_BAG_L1 ... Training model for up to 399.04s of the 599.1s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	-130.4612	 = Validation score   (root_mean_squared_error)
	192.76s	 = Training   runtime
	0.03s	 = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L1 ... Training model for up to 206.17s of the 406.24s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 70 due to low memory. Expected memory usage reduced from 63.55% -> 15.0% of available memory...
	-126.8015	 = Validation score   (root_mean_squared_error)
	0.26s	 = Training   runtime
	0.07s	 = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L1 ... Training model for up to 205.78s of the 405.85s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	-135.6753	 = Validation score   (root_mean_squared_error)
	56.1s	 = Training   runtime
	0.11s	 = Validation runtime
Fitting model: XGBoost_BAG_L1 ... Training model for up to 149.5s of the 349.57s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 389 MB
Estimated XGB model size: 1 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 391 MB
Estimated XGB model size: 3 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 391 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 4 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 390 MB
Estimated XGB model size: 3 MB
	-245.1743	 = Validation score   (root_mean_squared_error)
	0.17s	 = Training   runtime
	0.03s	 = Validation runtime
Fitting model: NeuralNetTorch_BAG_L1 ... Training model for up to 149.28s of the 349.35s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Ran out of time, stopping training early. (Stopping on epoch 38)
	Ran out of time, stopping training early. (Stopping on epoch 41)
	Ran out of time, stopping training early. (Stopping on epoch 43)
	Ran out of time, stopping training early. (Stopping on epoch 44)
	Ran out of time, stopping training early. (Stopping on epoch 47)
	Ran out of time, stopping training early. (Stopping on epoch 50)
	Ran out of time, stopping training early. (Stopping on epoch 55)
	Ran out of time, stopping training early. (Stopping on epoch 67)
	-139.9052	 = Validation score   (root_mean_squared_error)
	142.73s	 = Training   runtime
	0.14s	 = Validation runtime
Fitting model: LightGBMLarge_BAG_L1 ... Training model for up to 6.39s of the 206.46s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMLarge_BAG_L1 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L2 ... Training model for up to 360.0s of the 206.41s of remaining time.
	-84.1058	 = Validation score   (root_mean_squared_error)
	0.15s	 = Training   runtime
	0.0s	 = Validation runtime
Fitting 9 L2 models ...
Fitting model: LightGBMXT_BAG_L2 ... Training model for up to 206.26s of the 206.25s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMXT_BAG_L2 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: LightGBM_BAG_L2 ... Training model for up to 206.23s of the 206.23s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBM_BAG_L2 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: RandomForestMSE_BAG_L2 ... Training model for up to 206.21s of the 206.21s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 75 due to low memory. Expected memory usage reduced from 59.44% -> 15.0% of available memory...
	-54.1643	 = Validation score   (root_mean_squared_error)
	1.24s	 = Training   runtime
	0.08s	 = Validation runtime
Fitting model: CatBoost_BAG_L2 ... Training model for up to 204.82s of the 204.82s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	-55.3827	 = Validation score   (root_mean_squared_error)
	31.92s	 = Training   runtime
	0.02s	 = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L2 ... Training model for up to 172.85s of the 172.84s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 73 due to low memory. Expected memory usage reduced from 61.44% -> 15.0% of available memory...
	-54.3022	 = Validation score   (root_mean_squared_error)
	0.37s	 = Training   runtime
	0.08s	 = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L2 ... Training model for up to 172.35s of the 172.35s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	-52.281	 = Validation score   (root_mean_squared_error)
	57.07s	 = Training   runtime
	0.12s	 = Validation runtime
Fitting model: XGBoost_BAG_L2 ... Training model for up to 115.08s of the 115.07s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 398 MB
Estimated XGB model size: 1 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 396 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 396 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 398 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 398 MB
Estimated XGB model size: 1 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 397 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 397 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 397 MB
Estimated XGB model size: 8 MB
	-239.1839	 = Validation score   (root_mean_squared_error)
	0.26s	 = Training   runtime
	0.04s	 = Validation runtime
Fitting model: NeuralNetTorch_BAG_L2 ... Training model for up to 114.76s of the 114.75s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Ran out of time, stopping training early. (Stopping on epoch 30)
	Ran out of time, stopping training early. (Stopping on epoch 31)
	Ran out of time, stopping training early. (Stopping on epoch 32)
	Ran out of time, stopping training early. (Stopping on epoch 34)
	Ran out of time, stopping training early. (Stopping on epoch 35)
	Ran out of time, stopping training early. (Stopping on epoch 38)
	Ran out of time, stopping training early. (Stopping on epoch 42)
	Ran out of time, stopping training early. (Stopping on epoch 51)
	-58.4412	 = Validation score   (root_mean_squared_error)
	109.82s	 = Training   runtime
	0.15s	 = Validation runtime
Fitting model: LightGBMLarge_BAG_L2 ... Training model for up to 4.76s of the 4.76s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMLarge_BAG_L2 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L3 ... Training model for up to 360.0s of the 4.72s of remaining time.
	-51.0252	 = Validation score   (root_mean_squared_error)
	0.12s	 = Training   runtime
	0.0s	 = Validation runtime
AutoGluon training complete, total runtime = 595.41s ... Best model: "WeightedEnsemble_L3"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20220403_215705/")

Review AutoGluon's training run with ranking of models that did the best.

predictor.fit_summary()

*** Summary of fit() ***
Estimated performance of each model:
                     model   score_val  pred_time_val    fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  fit_order
0      WeightedEnsemble_L3  -51.025159       0.771097  451.308120                0.000255           0.119811            3       True         16
1   NeuralNetFastAI_BAG_L2  -52.280977       0.614861  449.583902                0.122451          57.067749            2       True         13
2   RandomForestMSE_BAG_L2  -54.164250       0.570976  393.755368                0.078566           1.239215            2       True         10
3     ExtraTreesMSE_BAG_L2  -54.302187       0.569826  392.881345                0.077416           0.365192            2       True         12
4          CatBoost_BAG_L2  -55.382727       0.515676  424.435747                0.023267          31.919594            2       True         11
5    NeuralNetTorch_BAG_L2  -58.441240       0.644520  502.331243                0.152110         109.815089            2       True         15
6    KNeighborsDist_BAG_L1  -84.105808       0.030788    0.016252                0.030788           0.016252            1       True          2
7      WeightedEnsemble_L2  -84.105808       0.031049    0.164693                0.000261           0.148441            2       True          9
8    KNeighborsUnif_BAG_L1 -101.506986       0.027615    0.040074                0.027615           0.040074            1       True          1
9   RandomForestMSE_BAG_L1 -119.081773       0.062411    0.431252                0.062411           0.431252            1       True          3
10    ExtraTreesMSE_BAG_L1 -126.801500       0.065241    0.261330                0.065241           0.261330            1       True          5
11         CatBoost_BAG_L1 -130.461205       0.031717  192.763849                0.031717         192.763849            1       True          4
12  NeuralNetFastAI_BAG_L1 -135.675261       0.108620   56.096116                0.108620          56.096116            1       True          6
13   NeuralNetTorch_BAG_L1 -139.905160       0.135293  142.733524                0.135293         142.733524            1       True          8
14          XGBoost_BAG_L2 -239.183912       0.532403  392.773360                0.039994           0.257206            2       True         14
15          XGBoost_BAG_L1 -245.174314       0.030725    0.173756                0.030725           0.173756            1       True          7
Number of models trained: 16
Types of models trained:
{'StackerEnsembleModel_CatBoost', 'StackerEnsembleModel_KNN', 'WeightedEnsembleModel', 'StackerEnsembleModel_TabularNeuralNetTorch', 'StackerEnsembleModel_XT', 'StackerEnsembleModel_RF', 'StackerEnsembleModel_XGBoost', 'StackerEnsembleModel_NNFastAiTabular'}
Bagging used: True  (with 8 folds)
Multi-layer stack-ensembling used: True  (with 3 levels)
Feature Metadata (Processed):
(raw dtype, special dtypes):
('float', [])                : 3 | ['temp', 'atemp', 'windspeed']
('int', [])                  : 3 | ['season', 'weather', 'humidity']
('int', ['bool'])            : 2 | ['holiday', 'workingday']
('int', ['datetime_as_int']) : 5 | ['datetime', 'datetime.year', 'datetime.month', 'datetime.day', 'datetime.dayofweek']
Plot summary of models saved to file: AutogluonModels/ag-20220403_215705/SummaryOfModels.html
*** End of fit() summary ***

{'model_types': {'KNeighborsUnif_BAG_L1': 'StackerEnsembleModel_KNN',
  'KNeighborsDist_BAG_L1': 'StackerEnsembleModel_KNN',
  'RandomForestMSE_BAG_L1': 'StackerEnsembleModel_RF',
  'CatBoost_BAG_L1': 'StackerEnsembleModel_CatBoost',
  'ExtraTreesMSE_BAG_L1': 'StackerEnsembleModel_XT',
  'NeuralNetFastAI_BAG_L1': 'StackerEnsembleModel_NNFastAiTabular',
  'XGBoost_BAG_L1': 'StackerEnsembleModel_XGBoost',
  'NeuralNetTorch_BAG_L1': 'StackerEnsembleModel_TabularNeuralNetTorch',
  'WeightedEnsemble_L2': 'WeightedEnsembleModel',
  'RandomForestMSE_BAG_L2': 'StackerEnsembleModel_RF',
  'CatBoost_BAG_L2': 'StackerEnsembleModel_CatBoost',
  'ExtraTreesMSE_BAG_L2': 'StackerEnsembleModel_XT',
  'NeuralNetFastAI_BAG_L2': 'StackerEnsembleModel_NNFastAiTabular',
  'XGBoost_BAG_L2': 'StackerEnsembleModel_XGBoost',
  'NeuralNetTorch_BAG_L2': 'StackerEnsembleModel_TabularNeuralNetTorch',
  'WeightedEnsemble_L3': 'WeightedEnsembleModel'},
 'model_performance': {'KNeighborsUnif_BAG_L1': -101.5069858001322,
  'KNeighborsDist_BAG_L1': -84.10580775218392,
  'RandomForestMSE_BAG_L1': -119.08177302978171,
  'CatBoost_BAG_L1': -130.46120460893425,
  'ExtraTreesMSE_BAG_L1': -126.80149965568127,
  'NeuralNetFastAI_BAG_L1': -135.67526066842626,
  'XGBoost_BAG_L1': -245.17431441781224,
  'NeuralNetTorch_BAG_L1': -139.90515963131176,
  'WeightedEnsemble_L2': -84.10580775218392,
  'RandomForestMSE_BAG_L2': -54.16425013512568,
  'CatBoost_BAG_L2': -55.38272690001613,
  'ExtraTreesMSE_BAG_L2': -54.302186838226554,
  'NeuralNetFastAI_BAG_L2': -52.28097726236311,
  'XGBoost_BAG_L2': -239.1839116086003,
  'NeuralNetTorch_BAG_L2': -58.441239557285215,
  'WeightedEnsemble_L3': -51.02515926002657},
 'model_best': 'WeightedEnsemble_L3',
 'model_paths': {'KNeighborsUnif_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/KNeighborsUnif_BAG_L1/',
  'KNeighborsDist_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/KNeighborsDist_BAG_L1/',
  'RandomForestMSE_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/RandomForestMSE_BAG_L1/',
  'CatBoost_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/CatBoost_BAG_L1/',
  'ExtraTreesMSE_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/ExtraTreesMSE_BAG_L1/',
  'NeuralNetFastAI_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/NeuralNetFastAI_BAG_L1/',
  'XGBoost_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/XGBoost_BAG_L1/',
  'NeuralNetTorch_BAG_L1': 'AutogluonModels/ag-20220403_215705/models/NeuralNetTorch_BAG_L1/',
  'WeightedEnsemble_L2': 'AutogluonModels/ag-20220403_215705/models/WeightedEnsemble_L2/',
  'RandomForestMSE_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/RandomForestMSE_BAG_L2/',
  'CatBoost_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/CatBoost_BAG_L2/',
  'ExtraTreesMSE_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/ExtraTreesMSE_BAG_L2/',
  'NeuralNetFastAI_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/NeuralNetFastAI_BAG_L2/',
  'XGBoost_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/XGBoost_BAG_L2/',
  'NeuralNetTorch_BAG_L2': 'AutogluonModels/ag-20220403_215705/models/NeuralNetTorch_BAG_L2/',
  'WeightedEnsemble_L3': 'AutogluonModels/ag-20220403_215705/models/WeightedEnsemble_L3/'},
 'model_fit_times': {'KNeighborsUnif_BAG_L1': 0.04007411003112793,
  'KNeighborsDist_BAG_L1': 0.01625204086303711,
  'RandomForestMSE_BAG_L1': 0.43125224113464355,
  'CatBoost_BAG_L1': 192.76384949684143,
  'ExtraTreesMSE_BAG_L1': 0.26133012771606445,
  'NeuralNetFastAI_BAG_L1': 56.096115827560425,
  'XGBoost_BAG_L1': 0.17375564575195312,
  'NeuralNetTorch_BAG_L1': 142.73352360725403,
  'WeightedEnsemble_L2': 0.14844107627868652,
  'RandomForestMSE_BAG_L2': 1.2392151355743408,
  'CatBoost_BAG_L2': 31.919593572616577,
  'ExtraTreesMSE_BAG_L2': 0.3651919364929199,
  'NeuralNetFastAI_BAG_L2': 57.0677490234375,
  'XGBoost_BAG_L2': 0.25720643997192383,
  'NeuralNetTorch_BAG_L2': 109.81508946418762,
  'WeightedEnsemble_L3': 0.1198110580444336},
 'model_pred_times': {'KNeighborsUnif_BAG_L1': 0.027614831924438477,
  'KNeighborsDist_BAG_L1': 0.030787944793701172,
  'RandomForestMSE_BAG_L1': 0.06241106986999512,
  'CatBoost_BAG_L1': 0.03171730041503906,
  'ExtraTreesMSE_BAG_L1': 0.06524085998535156,
  'NeuralNetFastAI_BAG_L1': 0.10861992835998535,
  'XGBoost_BAG_L1': 0.030724525451660156,
  'NeuralNetTorch_BAG_L1': 0.13529324531555176,
  'WeightedEnsemble_L2': 0.00026106834411621094,
  'RandomForestMSE_BAG_L2': 0.07856607437133789,
  'CatBoost_BAG_L2': 0.02326655387878418,
  'ExtraTreesMSE_BAG_L2': 0.07741594314575195,
  'NeuralNetFastAI_BAG_L2': 0.1224508285522461,
  'XGBoost_BAG_L2': 0.0399935245513916,
  'NeuralNetTorch_BAG_L2': 0.15211033821105957,
  'WeightedEnsemble_L3': 0.0002548694610595703},
 'num_bag_folds': 8,
 'max_stack_level': 3,
 'model_hyperparams': {'KNeighborsUnif_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'KNeighborsDist_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'RandomForestMSE_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'CatBoost_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'ExtraTreesMSE_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'NeuralNetFastAI_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'XGBoost_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'NeuralNetTorch_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'WeightedEnsemble_L2': {'use_orig_features': False,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'RandomForestMSE_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'CatBoost_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'ExtraTreesMSE_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'NeuralNetFastAI_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'XGBoost_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'NeuralNetTorch_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'WeightedEnsemble_L3': {'use_orig_features': False,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True}},
 'leaderboard':                      model   score_val  pred_time_val    fit_time  \
 0      WeightedEnsemble_L3  -51.025159       0.771097  451.308120   
 1   NeuralNetFastAI_BAG_L2  -52.280977       0.614861  449.583902   
 2   RandomForestMSE_BAG_L2  -54.164250       0.570976  393.755368   
 3     ExtraTreesMSE_BAG_L2  -54.302187       0.569826  392.881345   
 4          CatBoost_BAG_L2  -55.382727       0.515676  424.435747   
 5    NeuralNetTorch_BAG_L2  -58.441240       0.644520  502.331243   
 6    KNeighborsDist_BAG_L1  -84.105808       0.030788    0.016252   
 7      WeightedEnsemble_L2  -84.105808       0.031049    0.164693   
 8    KNeighborsUnif_BAG_L1 -101.506986       0.027615    0.040074   
 9   RandomForestMSE_BAG_L1 -119.081773       0.062411    0.431252   
 10    ExtraTreesMSE_BAG_L1 -126.801500       0.065241    0.261330   
 11         CatBoost_BAG_L1 -130.461205       0.031717  192.763849   
 12  NeuralNetFastAI_BAG_L1 -135.675261       0.108620   56.096116   
 13   NeuralNetTorch_BAG_L1 -139.905160       0.135293  142.733524   
 14          XGBoost_BAG_L2 -239.183912       0.532403  392.773360   
 15          XGBoost_BAG_L1 -245.174314       0.030725    0.173756   
 
     pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \
 0                 0.000255           0.119811            3       True   
 1                 0.122451          57.067749            2       True   
 2                 0.078566           1.239215            2       True   
 3                 0.077416           0.365192            2       True   
 4                 0.023267          31.919594            2       True   
 5                 0.152110         109.815089            2       True   
 6                 0.030788           0.016252            1       True   
 7                 0.000261           0.148441            2       True   
 8                 0.027615           0.040074            1       True   
 9                 0.062411           0.431252            1       True   
 10                0.065241           0.261330            1       True   
 11                0.031717         192.763849            1       True   
 12                0.108620          56.096116            1       True   
 13                0.135293         142.733524            1       True   
 14                0.039994           0.257206            2       True   
 15                0.030725           0.173756            1       True   
 
     fit_order  
 0          16  
 1          13  
 2          10  
 3          12  
 4          11  
 5          15  
 6           2  
 7           9  
 8           1  
 9           3  
 10          5  
 11          4  
 12          6  
 13          8  
 14         14  
 15          7  }

# Output the fit summary of the training run
predictor.leaderboard(silent=True).plot(kind="bar", x="model", y="score_val")

<AxesSubplot:xlabel='model'>

Create predictions from test dataset

predictions = predictor.predict(test)
predictions.head()

0    25.395908
1    40.721481
2    46.926262
3    48.073616
4    49.973679
Name: count, dtype: float32

NOTE: Kaggle will reject the submission if we don't set everything to be > 0.

# Describe the `predictions` series to see if there are any negative values
predictions.describe()

count    6493.000000
mean       99.825455
std        88.543869
min         1.512626
25%        20.131336
50%        67.336700
75%       169.070740
max       356.121338
Name: count, dtype: float64

# How many negative values do we have?
predictions.lt(0).sum()

0

# Set them to zero
predictions.clip(lower=0)

0        25.395908
1        40.721481
2        46.926262
3        48.073616
4        49.973679
           ...    
6488    170.985306
6489    170.989395
6490    157.917023
6491    154.561798
6492    148.553497
Name: count, Length: 6493, dtype: float32

Set predictions to submission dataframe, save, and submit

submission["count"] = predictions
# Round count column values
submission["count"] = submission["count"].apply(lambda x: round(x))
submission.to_csv("submission.csv", index=False)

submission.head()

	datetime	count
0	2011-01-20 00:00:00	25
1	2011-01-20 01:00:00	41
2	2011-01-20 02:00:00	47
3	2011-01-20 03:00:00	48
4	2011-01-20 04:00:00	50

!kaggle competitions submit -c bike-sharing-demand -f submission.csv -m "first raw submission"

100%|████████████████████████████████████████| 148k/148k [00:03<00:00, 47.1kB/s]
Successfully submitted to Bike Sharing Demand

View submission via the command line or in the web browser under the competition's page - My Submissions

!kaggle competitions submissions -c bike-sharing-demand | tail -n +1 | head -n 6

fileName        date                 description           status    publicScore  privateScore  
--------------  -------------------  --------------------  --------  -----------  ------------  
submission.csv  2022-04-03 22:23:11  first raw submission  complete  1.80656      1.80656       

Initial score of 1.80656

Step 4: Exploratory Data Analysis and Creating an additional feature

• Any additional feature will do, but a great suggestion would be to separate out the datetime into hour, day, or month parts.

from pandas_profiling import ProfileReport
profile = ProfileReport(train, title="Pandas Profiling Report")
profile

# Create a histogram of all features to show the distribution of each one relative to the data. 
# This is part of the exploritory data analysis
train = train.drop(['casual','registered'], axis=1)
train.hist(figsize=(15, 10));

Datetime

# time series plot for demand
train.plot(x="datetime", y="count", figsize=(15,6))

<AxesSubplot:xlabel='datetime'>

# separate out the datetime into hour, day, month parts
train["year"] = train["datetime"].dt.year
train["month"] = train["datetime"].dt.month
train["day"] = train["datetime"].dt.day
train["day_of_week"] = train["datetime"].dt.day_of_week
train["hour"] = train["datetime"].dt.hour

# add same teatures in test set
test["year"] = test["datetime"].dt.year
test["month"] = test["datetime"].dt.month
test["day"] = test["datetime"].dt.day
test["day_of_week"] = test["datetime"].dt.day_of_week
test["hour"] = test["datetime"].dt.hour

train.head()

	datetime	season	holiday	workingday	weather	temp	atemp	humidity	windspeed	count	year	month	day	day_of_week	hour
0	2011-01-01 00:00:00	1	0	0	1	9.84	14.395	81	0.0	16	2011	1	1	5	0
1	2011-01-01 01:00:00	1	0	0	1	9.02	13.635	80	0.0	40	2011	1	1	5	1
2	2011-01-01 02:00:00	1	0	0	1	9.02	13.635	80	0.0	32	2011	1	1	5	2
3	2011-01-01 03:00:00	1	0	0	1	9.84	14.395	75	0.0	13	2011	1	1	5	3
4	2011-01-01 04:00:00	1	0	0	1	9.84	14.395	75	0.0	1	2011	1	1	5	4

# time series plot for a paticular month: 2011-4
train[ (train["year"]==2011) & (train["month"]==4) ].plot(x="datetime", y="count", figsize=(15,6))

<AxesSubplot:xlabel='datetime'>

# time series plot for a paticular week: 2011-4-1 - 2011-4-8
train[ 
    (train["datetime"]>='2011-4-1') & 
    (train["datetime"]<='2011-4-8')
].plot(x="datetime", y="count", figsize=(15,6))

<AxesSubplot:xlabel='datetime'>

Hour

# plot for all working days
plt.figure(figsize=(15,6))
sns.barplot(x="hour",y="count",data=train[train["workingday"]==1], color=sns.color_palette()[0], ci=None)

<AxesSubplot:xlabel='hour', ylabel='count'>

# plot for all weekend
plt.figure(figsize=(15,6))
sns.barplot(x="hour",y="count",data=train[train["workingday"]==0], color=sns.color_palette()[0], ci=None)

<AxesSubplot:xlabel='hour', ylabel='count'>

Temprature

# compute the correlation between "temp" and "atemp"
train[["temp", "atemp"]].corr()

	temp	atemp
temp	1.000000	0.984948
atemp	0.984948	1.000000

# we choose temp rather than atemp becasue the distribution "feel-like" temp has more bias
# plot for temp
plt.figure(figsize=(15,6))
sns.barplot(x="temp",y="count", data=train, color=sns.color_palette()[0], ci=None)
plt.xticks(rotation = 90);

# create new feature: temp_category
def get_temp_category(temp):
    if temp <= 13:
        return 0 # low temp
    elif (temp > 13) and (temp <= 28 ):
        return 1 # mild temp
    elif (temp > 28):
        return 2 # hot temp

train['temp_category'] = train['temp'].apply(get_temp_category)
test['temp_category'] = test['temp'].apply(get_temp_category)

Humidity

# plot for humidity
plt.figure(figsize=(15,6))
sns.barplot(x="humidity", y="count", data=train, color=sns.color_palette()[0], ci=None)
plt.xticks(rotation = 90);

# create new feature: humidity_category
def get_humid_category(humid):
    if humid <= 15:
        return 0 # low temp
    elif (humid > 15) and (humid <= 46 ):
        return 1 # mild temp
    elif (humid > 46) and (humid < 75 ):
        return 2 # hot temp
    else:
        return 3

train['humid_category'] = train['humidity'].apply(get_humid_category)
test['humid_category'] = test['humidity'].apply(get_humid_category)

Windspeed

# the distribution of windspeed shows there are outliers
sns.boxplot(data=train, y="windspeed",orient="v")

<AxesSubplot:ylabel='windspeed'>

train = train[train["windspeed"]-train["windspeed"].mean() <=(3*train["windspeed"].std())]

# plot for windspeed
plt.figure(figsize=(15,6))
sns.barplot(x="windspeed", y="count", data=train, color=sns.color_palette()[0], ci=None)
plt.xticks(rotation = 90);

# create new feature: wind_category
def get_wind_category(wind):
    if wind <= 10:
        return 0 # low wind
    elif (wind > 10) and (wind <= 20 ):
        return 1 # mild wind
    elif (wind > 20):
        return 2 # high temp

train['wind_category'] = train['windspeed'].apply(get_wind_category)
test['wind_category'] = test['windspeed'].apply(get_wind_category)

# drop unnecessary features
train.drop(['datetime', 'temp', 'windspeed', 'humidity', 'atemp'], axis=1, inplace=True)
test.drop(['datetime', 'temp', 'windspeed', 'humidity', 'atemp'], axis=1, inplace=True)

# plot correlations between features
plt.figure(figsize=(10, 8), dpi=80)
sns.heatmap(train.drop(['count'], axis=1).corr(), annot=True)
plt.show()

# removing season column as it's having very high correlation with month
train.drop(['season', 'day_of_week'], axis=1, inplace=True)
test.drop(['season', 'day_of_week'], axis=1, inplace=True)

Make category types for these so models know they are not just numbers

• AutoGluon originally sees these as ints, but in reality they are int representations of a category.
• Setting the dtype to category will classify these as categories in AutoGluon.

train["weather"] = train["weather"].astype('category')
train["temp_category"] = train["temp_category"].astype('category')
train["wind_category"] = train["wind_category"].astype('category')
train["humid_category"] = train["humid_category"].astype('category')
train["holiday"] = train["holiday"].astype('category')
train["workingday"] = train["workingday"].astype('category')
train["hour"] = train["hour"].astype('category')
train["month"] = train["month"].astype('category')

test["weather"] = test["weather"].astype('category')
test["temp_category"] = test["temp_category"].astype('category')
test["wind_category"] = test["wind_category"].astype('category')
test["humid_category"] = test["humid_category"].astype('category')
test["holiday"] = test["holiday"].astype('category')
test["workingday"] = test["workingday"].astype('category')
test["hour"] = test["hour"].astype('category')
test["month"] = test["month"].astype('category')

train.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 10819 entries, 0 to 10885
Data columns (total 11 columns):
 #   Column          Non-Null Count  Dtype   
---  ------          --------------  -----   
 0   holiday         10819 non-null  category
 1   workingday      10819 non-null  category
 2   weather         10819 non-null  category
 3   count           10819 non-null  int64   
 4   year            10819 non-null  int64   
 5   month           10819 non-null  category
 6   day             10819 non-null  int64   
 7   hour            10819 non-null  category
 8   temp_category   10819 non-null  category
 9   humid_category  10819 non-null  category
 10  wind_category   10819 non-null  category
dtypes: category(8), int64(3)
memory usage: 682.7 KB

# View are new feature
train.head()

	holiday	workingday	weather	count	year	month	day	hour	temp_category	humid_category	wind_category
0	0	0	1	16	2011	1	1	0	0	3	0
1	0	0	1	40	2011	1	1	1	0	3	0
2	0	0	1	32	2011	1	1	2	0	3	0
3	0	0	1	13	2011	1	1	3	0	3	0
4	0	0	1	1	2011	1	1	4	0	3	0

# View histogram of all features again now with the hour feature
train.hist(figsize=(15,10));

Step 5: Rerun the model with the same settings as before, just with more features

predictor_new_features = TabularPredictor(
    label="count", 
    problem_type="regression", 
    eval_metric="root_mean_squared_error"
).fit(
    train_data=train,
    time_limit=600,
    presets="best_quality"
)

No path specified. Models will be saved in: "AutogluonModels/ag-20220404_171935/"
Presets specified: ['best_quality']
Beginning AutoGluon training ... Time limit = 600s
AutoGluon will save models to "AutogluonModels/ag-20220404_171935/"
AutoGluon Version:  0.4.0
Python Version:     3.9.7
Operating System:   Darwin
Train Data Rows:    10819
Train Data Columns: 10
Label Column: count
Preprocessing data ...
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
	Available Memory:                    399.34 MB
	Train Data (Original)  Memory Usage: 0.26 MB (0.1% of available memory)
	Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
	Stage 1 Generators:
		Fitting AsTypeFeatureGenerator...
			Note: Converting 3 features to boolean dtype as they only contain 2 unique values.
	Stage 2 Generators:
		Fitting FillNaFeatureGenerator...
	Stage 3 Generators:
		Fitting IdentityFeatureGenerator...
		Fitting CategoryFeatureGenerator...
			Fitting CategoryMemoryMinimizeFeatureGenerator...
	Stage 4 Generators:
		Fitting DropUniqueFeatureGenerator...
	Types of features in original data (raw dtype, special dtypes):
		('category', []) : 8 | ['holiday', 'workingday', 'weather', 'month', 'hour', ...]
		('int', [])      : 2 | ['year', 'day']
	Types of features in processed data (raw dtype, special dtypes):
		('category', [])  : 6 | ['weather', 'month', 'hour', 'temp_category', 'humid_category', ...]
		('int', [])       : 1 | ['day']
		('int', ['bool']) : 3 | ['holiday', 'workingday', 'year']
	0.1s = Fit runtime
	10 features in original data used to generate 10 features in processed data.
	Train Data (Processed) Memory Usage: 0.19 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.07s ...
AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error'
	To change this, specify the eval_metric parameter of Predictor()
AutoGluon will fit 2 stack levels (L1 to L2) ...
Fitting 11 L1 models ...
Fitting model: KNeighborsUnif_BAG_L1 ... Training model for up to 399.85s of the 599.93s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
	-246.8409	 = Validation score   (root_mean_squared_error)
	2.61s	 = Training   runtime
	0.04s	 = Validation runtime
Fitting model: KNeighborsDist_BAG_L1 ... Training model for up to 397.16s of the 597.23s of remaining time.
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
	-246.8409	 = Validation score   (root_mean_squared_error)
	0.01s	 = Training   runtime
	0.05s	 = Validation runtime
Fitting model: LightGBMXT_BAG_L1 ... Training model for up to 397.09s of the 597.16s of remaining time.
	Will use sequential fold fitting strategy because Darwin OS does not yet support parallel folding.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMXT_BAG_L1 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: LightGBM_BAG_L1 ... Training model for up to 396.47s of the 596.55s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBM_BAG_L1 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: RandomForestMSE_BAG_L1 ... Training model for up to 396.46s of the 596.53s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 65 due to low memory. Expected memory usage reduced from 68.23% -> 15.0% of available memory...
	-44.4183	 = Validation score   (root_mean_squared_error)
	0.28s	 = Training   runtime
	0.06s	 = Validation runtime
Fitting model: CatBoost_BAG_L1 ... Training model for up to 396.07s of the 596.14s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Ran out of time, early stopping on iteration 6141.
	Ran out of time, early stopping on iteration 6324.
	Ran out of time, early stopping on iteration 6676.
	Ran out of time, early stopping on iteration 7916.
	Ran out of time, early stopping on iteration 8392.
	Ran out of time, early stopping on iteration 9546.
	-45.5806	 = Validation score   (root_mean_squared_error)
	377.48s	 = Training   runtime
	0.1s	 = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L1 ... Training model for up to 18.26s of the 218.34s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 64 due to low memory. Expected memory usage reduced from 70.19% -> 15.0% of available memory...
	-45.6605	 = Validation score   (root_mean_squared_error)
	0.21s	 = Training   runtime
	0.07s	 = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L1 ... Training model for up to 17.92s of the 218.0s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Ran out of time, stopping training early. (Stopping on epoch 3)
	Time limit exceeded... Skipping NeuralNetFastAI_BAG_L1.
Fitting model: XGBoost_BAG_L1 ... Training model for up to 15.33s of the 215.4s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 338 MB
Estimated XGB model size: 3 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 330 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 323 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 324 MB
Estimated XGB model size: 0 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 325 MB
Estimated XGB model size: 0 MB
	-243.6172	 = Validation score   (root_mean_squared_error)
	0.4s	 = Training   runtime
	0.05s	 = Validation runtime
Fitting model: NeuralNetTorch_BAG_L1 ... Training model for up to 14.86s of the 214.93s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Ran out of time, stopping training early. (Stopping on epoch 3)
	Ran out of time, stopping training early. (Stopping on epoch 3)
	Ran out of time, stopping training early. (Stopping on epoch 4)
	Ran out of time, stopping training early. (Stopping on epoch 4)
	Ran out of time, stopping training early. (Stopping on epoch 5)
	-102.5411	 = Validation score   (root_mean_squared_error)
	13.79s	 = Training   runtime
	0.19s	 = Validation runtime
Fitting model: LightGBMLarge_BAG_L1 ... Training model for up to 0.85s of the 200.93s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMLarge_BAG_L1 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L2 ... Training model for up to 360.0s of the 200.9s of remaining time.
	-41.9413	 = Validation score   (root_mean_squared_error)
	0.14s	 = Training   runtime
	0.0s	 = Validation runtime
Fitting 9 L2 models ...
Fitting model: LightGBMXT_BAG_L2 ... Training model for up to 200.75s of the 200.75s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMXT_BAG_L2 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: LightGBM_BAG_L2 ... Training model for up to 200.73s of the 200.73s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBM_BAG_L2 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Fitting model: RandomForestMSE_BAG_L2 ... Training model for up to 200.71s of the 200.71s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 66 due to low memory. Expected memory usage reduced from 67.91% -> 15.0% of available memory...
	-43.7209	 = Validation score   (root_mean_squared_error)
	0.77s	 = Training   runtime
	0.07s	 = Validation runtime
Fitting model: CatBoost_BAG_L2 ... Training model for up to 199.8s of the 199.79s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	-42.3145	 = Validation score   (root_mean_squared_error)
	20.97s	 = Training   runtime
	0.04s	 = Validation runtime
Fitting model: ExtraTreesMSE_BAG_L2 ... Training model for up to 178.75s of the 178.75s of remaining time.
	Warning: Reducing model 'n_estimators' from 300 -> 64 due to low memory. Expected memory usage reduced from 69.86% -> 15.0% of available memory...
	-42.7483	 = Validation score   (root_mean_squared_error)
	0.28s	 = Training   runtime
	0.07s	 = Validation runtime
Fitting model: NeuralNetFastAI_BAG_L2 ... Training model for up to 178.35s of the 178.34s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
No improvement since epoch 4: early stopping
No improvement since epoch 6: early stopping
No improvement since epoch 3: early stopping
No improvement since epoch 4: early stopping
No improvement since epoch 2: early stopping
	-42.5218	 = Validation score   (root_mean_squared_error)
	58.3s	 = Training   runtime
	0.16s	 = Validation runtime
Fitting model: XGBoost_BAG_L2 ... Training model for up to 119.8s of the 119.79s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 318 MB
Estimated XGB model size: 21 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 318 MB
Estimated XGB model size: 9 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 319 MB
Estimated XGB model size: 7 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 317 MB
Estimated XGB model size: 4 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 316 MB
Estimated XGB model size: 7 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 321 MB
Estimated XGB model size: 3 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 322 MB
Estimated XGB model size: 2 MB
Warning: Large XGB model size may cause OOM error if training continues
Available Memory: 322 MB
Estimated XGB model size: 0 MB
	-238.0238	 = Validation score   (root_mean_squared_error)
	0.5s	 = Training   runtime
	0.05s	 = Validation runtime
Fitting model: NeuralNetTorch_BAG_L2 ... Training model for up to 119.22s of the 119.22s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Ran out of time, stopping training early. (Stopping on epoch 26)
	Ran out of time, stopping training early. (Stopping on epoch 27)
	Ran out of time, stopping training early. (Stopping on epoch 28)
	Ran out of time, stopping training early. (Stopping on epoch 29)
	Ran out of time, stopping training early. (Stopping on epoch 31)
	Ran out of time, stopping training early. (Stopping on epoch 33)
	Ran out of time, stopping training early. (Stopping on epoch 36)
	Ran out of time, stopping training early. (Stopping on epoch 45)
	-43.8899	 = Validation score   (root_mean_squared_error)
	113.69s	 = Training   runtime
	0.23s	 = Validation runtime
Fitting model: LightGBMLarge_BAG_L2 ... Training model for up to 5.27s of the 5.26s of remaining time.
	Fitting 8 child models (S1F1 - S1F8) | Fitting with SequentialLocalFoldFittingStrategy
	Warning: Exception caused LightGBMLarge_BAG_L2 to fail during training (ImportError)... Skipping this model.
		`import lightgbm` failed. If you are using Mac OSX, Please try 'brew install libomp'. Detailed info: dlopen(/Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/wenkaiwu/opt/anaconda3/lib/python3.9/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Completed 1/20 k-fold bagging repeats ...
Fitting model: WeightedEnsemble_L3 ... Training model for up to 360.0s of the 5.21s of remaining time.
	-41.6628	 = Validation score   (root_mean_squared_error)
	0.12s	 = Training   runtime
	0.0s	 = Validation runtime
AutoGluon training complete, total runtime = 594.92s ... Best model: "WeightedEnsemble_L3"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20220404_171935/")

predictor_new_features.fit_summary()

*** Summary of fit() ***
Estimated performance of each model:
                     model   score_val  pred_time_val    fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  fit_order
0      WeightedEnsemble_L3  -41.662787       1.137516  588.921629                0.000282           0.122048            3       True         15
1      WeightedEnsemble_L2  -41.941318       0.232135  378.103965                0.000266           0.138094            2       True          8
2          CatBoost_BAG_L2  -42.314486       0.601931  415.760942                0.044700          20.974016            2       True         10
3   NeuralNetFastAI_BAG_L2  -42.521774       0.715616  453.085581                0.158386          58.298656            2       True         12
4     ExtraTreesMSE_BAG_L2  -42.748345       0.627164  395.070385                0.069933           0.283460            2       True         11
5   RandomForestMSE_BAG_L2  -43.720860       0.630273  395.556621                0.073042           0.769696            2       True          9
6    NeuralNetTorch_BAG_L2  -43.889909       0.791174  508.473753                0.233943         113.686828            2       True         14
7   RandomForestMSE_BAG_L1  -44.418272       0.064314    0.276485                0.064314           0.276485            1       True          3
8          CatBoost_BAG_L1  -45.580553       0.101993  377.483847                0.101993         377.483847            1       True          4
9     ExtraTreesMSE_BAG_L1  -45.660535       0.065563    0.205539                0.065563           0.205539            1       True          5
10   NeuralNetTorch_BAG_L1 -102.541134       0.185994   13.794680                0.185994          13.794680            1       True          7
11          XGBoost_BAG_L2 -238.023805       0.609681  395.291189                0.052450           0.504264            2       True         13
12          XGBoost_BAG_L1 -243.617185       0.049562    0.403421                0.049562           0.403421            1       True          6
13   KNeighborsUnif_BAG_L1 -246.840883       0.042858    2.608711                0.042858           2.608711            1       True          1
14   KNeighborsDist_BAG_L1 -246.840883       0.046947    0.014242                0.046947           0.014242            1       True          2
Number of models trained: 15
Types of models trained:
{'WeightedEnsembleModel', 'StackerEnsembleModel_KNN', 'StackerEnsembleModel_XT', 'StackerEnsembleModel_CatBoost', 'StackerEnsembleModel_XGBoost', 'StackerEnsembleModel_NNFastAiTabular', 'StackerEnsembleModel_TabularNeuralNetTorch', 'StackerEnsembleModel_RF'}
Bagging used: True  (with 8 folds)
Multi-layer stack-ensembling used: True  (with 3 levels)
Feature Metadata (Processed):
(raw dtype, special dtypes):
('category', [])  : 6 | ['weather', 'month', 'hour', 'temp_category', 'humid_category', ...]
('int', [])       : 1 | ['day']
('int', ['bool']) : 3 | ['holiday', 'workingday', 'year']
Plot summary of models saved to file: AutogluonModels/ag-20220404_171935/SummaryOfModels.html
*** End of fit() summary ***

{'model_types': {'KNeighborsUnif_BAG_L1': 'StackerEnsembleModel_KNN',
  'KNeighborsDist_BAG_L1': 'StackerEnsembleModel_KNN',
  'RandomForestMSE_BAG_L1': 'StackerEnsembleModel_RF',
  'CatBoost_BAG_L1': 'StackerEnsembleModel_CatBoost',
  'ExtraTreesMSE_BAG_L1': 'StackerEnsembleModel_XT',
  'XGBoost_BAG_L1': 'StackerEnsembleModel_XGBoost',
  'NeuralNetTorch_BAG_L1': 'StackerEnsembleModel_TabularNeuralNetTorch',
  'WeightedEnsemble_L2': 'WeightedEnsembleModel',
  'RandomForestMSE_BAG_L2': 'StackerEnsembleModel_RF',
  'CatBoost_BAG_L2': 'StackerEnsembleModel_CatBoost',
  'ExtraTreesMSE_BAG_L2': 'StackerEnsembleModel_XT',
  'NeuralNetFastAI_BAG_L2': 'StackerEnsembleModel_NNFastAiTabular',
  'XGBoost_BAG_L2': 'StackerEnsembleModel_XGBoost',
  'NeuralNetTorch_BAG_L2': 'StackerEnsembleModel_TabularNeuralNetTorch',
  'WeightedEnsemble_L3': 'WeightedEnsembleModel'},
 'model_performance': {'KNeighborsUnif_BAG_L1': -246.84088280283666,
  'KNeighborsDist_BAG_L1': -246.84088280283666,
  'RandomForestMSE_BAG_L1': -44.41827240398544,
  'CatBoost_BAG_L1': -45.58055346129427,
  'ExtraTreesMSE_BAG_L1': -45.66053480825233,
  'XGBoost_BAG_L1': -243.6171849564755,
  'NeuralNetTorch_BAG_L1': -102.54113388241504,
  'WeightedEnsemble_L2': -41.94131756111479,
  'RandomForestMSE_BAG_L2': -43.720860491299774,
  'CatBoost_BAG_L2': -42.314486219830165,
  'ExtraTreesMSE_BAG_L2': -42.74834515629869,
  'NeuralNetFastAI_BAG_L2': -42.52177351470511,
  'XGBoost_BAG_L2': -238.0238054930029,
  'NeuralNetTorch_BAG_L2': -43.88990930176027,
  'WeightedEnsemble_L3': -41.662786751754},
 'model_best': 'WeightedEnsemble_L3',
 'model_paths': {'KNeighborsUnif_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/KNeighborsUnif_BAG_L1/',
  'KNeighborsDist_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/KNeighborsDist_BAG_L1/',
  'RandomForestMSE_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/RandomForestMSE_BAG_L1/',
  'CatBoost_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/CatBoost_BAG_L1/',
  'ExtraTreesMSE_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/ExtraTreesMSE_BAG_L1/',
  'XGBoost_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/XGBoost_BAG_L1/',
  'NeuralNetTorch_BAG_L1': 'AutogluonModels/ag-20220404_171935/models/NeuralNetTorch_BAG_L1/',
  'WeightedEnsemble_L2': 'AutogluonModels/ag-20220404_171935/models/WeightedEnsemble_L2/',
  'RandomForestMSE_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/RandomForestMSE_BAG_L2/',
  'CatBoost_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/CatBoost_BAG_L2/',
  'ExtraTreesMSE_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/ExtraTreesMSE_BAG_L2/',
  'NeuralNetFastAI_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/NeuralNetFastAI_BAG_L2/',
  'XGBoost_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/XGBoost_BAG_L2/',
  'NeuralNetTorch_BAG_L2': 'AutogluonModels/ag-20220404_171935/models/NeuralNetTorch_BAG_L2/',
  'WeightedEnsemble_L3': 'AutogluonModels/ag-20220404_171935/models/WeightedEnsemble_L3/'},
 'model_fit_times': {'KNeighborsUnif_BAG_L1': 2.608711004257202,
  'KNeighborsDist_BAG_L1': 0.014241933822631836,
  'RandomForestMSE_BAG_L1': 0.27648472785949707,
  'CatBoost_BAG_L1': 377.48384737968445,
  'ExtraTreesMSE_BAG_L1': 0.20553922653198242,
  'XGBoost_BAG_L1': 0.40342116355895996,
  'NeuralNetTorch_BAG_L1': 13.794679880142212,
  'WeightedEnsemble_L2': 0.1380939483642578,
  'RandomForestMSE_BAG_L2': 0.7696959972381592,
  'CatBoost_BAG_L2': 20.974016427993774,
  'ExtraTreesMSE_BAG_L2': 0.2834599018096924,
  'NeuralNetFastAI_BAG_L2': 58.29865550994873,
  'XGBoost_BAG_L2': 0.5042638778686523,
  'NeuralNetTorch_BAG_L2': 113.68682765960693,
  'WeightedEnsemble_L3': 0.12204790115356445},
 'model_pred_times': {'KNeighborsUnif_BAG_L1': 0.042858123779296875,
  'KNeighborsDist_BAG_L1': 0.04694700241088867,
  'RandomForestMSE_BAG_L1': 0.06431388854980469,
  'CatBoost_BAG_L1': 0.10199284553527832,
  'ExtraTreesMSE_BAG_L1': 0.06556272506713867,
  'XGBoost_BAG_L1': 0.04956173896789551,
  'NeuralNetTorch_BAG_L1': 0.18599438667297363,
  'WeightedEnsemble_L2': 0.0002658367156982422,
  'RandomForestMSE_BAG_L2': 0.07304215431213379,
  'CatBoost_BAG_L2': 0.044699907302856445,
  'ExtraTreesMSE_BAG_L2': 0.06993293762207031,
  'NeuralNetFastAI_BAG_L2': 0.1583855152130127,
  'XGBoost_BAG_L2': 0.05245041847229004,
  'NeuralNetTorch_BAG_L2': 0.23394322395324707,
  'WeightedEnsemble_L3': 0.0002818107604980469},
 'num_bag_folds': 8,
 'max_stack_level': 3,
 'model_hyperparams': {'KNeighborsUnif_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'KNeighborsDist_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'RandomForestMSE_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'CatBoost_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'ExtraTreesMSE_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'XGBoost_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'NeuralNetTorch_BAG_L1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'WeightedEnsemble_L2': {'use_orig_features': False,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'RandomForestMSE_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'CatBoost_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'ExtraTreesMSE_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'NeuralNetFastAI_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'XGBoost_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'NeuralNetTorch_BAG_L2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'WeightedEnsemble_L3': {'use_orig_features': False,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True}},
 'leaderboard':                      model   score_val  pred_time_val    fit_time  \
 0      WeightedEnsemble_L3  -41.662787       1.137516  588.921629   
 1      WeightedEnsemble_L2  -41.941318       0.232135  378.103965   
 2          CatBoost_BAG_L2  -42.314486       0.601931  415.760942   
 3   NeuralNetFastAI_BAG_L2  -42.521774       0.715616  453.085581   
 4     ExtraTreesMSE_BAG_L2  -42.748345       0.627164  395.070385   
 5   RandomForestMSE_BAG_L2  -43.720860       0.630273  395.556621   
 6    NeuralNetTorch_BAG_L2  -43.889909       0.791174  508.473753   
 7   RandomForestMSE_BAG_L1  -44.418272       0.064314    0.276485   
 8          CatBoost_BAG_L1  -45.580553       0.101993  377.483847   
 9     ExtraTreesMSE_BAG_L1  -45.660535       0.065563    0.205539   
 10   NeuralNetTorch_BAG_L1 -102.541134       0.185994   13.794680   
 11          XGBoost_BAG_L2 -238.023805       0.609681  395.291189   
 12          XGBoost_BAG_L1 -243.617185       0.049562    0.403421   
 13   KNeighborsUnif_BAG_L1 -246.840883       0.042858    2.608711   
 14   KNeighborsDist_BAG_L1 -246.840883       0.046947    0.014242   
 
     pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \
 0                 0.000282           0.122048            3       True   
 1                 0.000266           0.138094            2       True   
 2                 0.044700          20.974016            2       True   
 3                 0.158386          58.298656            2       True   
 4                 0.069933           0.283460            2       True   
 5                 0.073042           0.769696            2       True   
 6                 0.233943         113.686828            2       True   
 7                 0.064314           0.276485            1       True   
 8                 0.101993         377.483847            1       True   
 9                 0.065563           0.205539            1       True   
 10                0.185994          13.794680            1       True   
 11                0.052450           0.504264            2       True   
 12                0.049562           0.403421            1       True   
 13                0.042858           2.608711            1       True   
 14                0.046947           0.014242            1       True   
 
     fit_order  
 0          15  
 1           8  
 2          10  
 3          12  
 4          11  
 5           9  
 6          14  
 7           3  
 8           4  
 9           5  
 10          7  
 11         13  
 12          6  
 13          1  
 14          2  }

# Output the fit summary of the training run
predictor_new_features.leaderboard(silent=True).plot(kind="bar", x="model", y="score_val")

<AxesSubplot:xlabel='model'>

# Remember to set all negative values to zero
predictions = predictor_new_features.predict(test)
predictions = predictions.clip(lower=0)

# Same submitting predictions
submission_new_features = pd.read_csv('data/sampleSubmission.csv', parse_dates=['datetime'])
submission_new_features["count"] = predictions
submission_new_features['count'] = submission_new_features['count'].apply(lambda x: round(x))

submission_new_features.to_csv("submission_new_features.csv", index=False)
submission_new_features.head()

	datetime	count
0	2011-01-20 00:00:00	15
1	2011-01-20 01:00:00	6
2	2011-01-20 02:00:00	3
3	2011-01-20 03:00:00	4
4	2011-01-20 04:00:00	3

!kaggle competitions submit -c bike-sharing-demand -f submission_new_features.csv -m "new features"

100%|████████████████████████████████████████| 149k/149k [00:02<00:00, 72.5kB/s]
Successfully submitted to Bike Sharing Demand

!kaggle competitions submissions -c bike-sharing-demand | tail -n +1 | head -n 6

fileName                     date                 description           status    publicScore  privateScore  
---------------------------  -------------------  --------------------  --------  -----------  ------------  
submission_new_features.csv  2022-04-04 17:33:48  new features          complete  0.50239      0.50239       
submission.csv               2022-04-03 22:23:11  first raw submission  complete  1.80656      1.80656       

New Score of 0.50239

Step 6: Hyper parameter optimization

• There are many options for hyper parameter optimization.
• Options are to change the AutoGluon higher level parameters or the individual model hyperparameters.
• The hyperparameters of the models themselves that are in AutoGluon. Those need the hyperparameter and hyperparameter_tune_kwargs arguments.

import autogluon.core as ag

nn_options = {
    'num_epochs': 10,
    'learning_rate': ag.space.Real(1e-4, 1e-2, default=5e-4, log=True),
    'activation': ag.space.Categorical('relu', 'softrelu', 'tanh'),
    'layers': ag.space.Categorical([100],[1000],[200,100],[300,200,100]),
    'dropout_prob': ag.space.Real(0.0, 0.5, default=0.1),
}

gbm_options = {
    'num_boost_round': 100,
    'num_leaves': ag.space.Int(lower=26, upper=66, default=36)
}

hyperparameters = {
    'GBM': gbm_options,
    'CAT': {'iterations': 10000},
    'RF': {'n_estimators': 300},
    'XT': {'n_estimators': 300}
    # 'NN': nn_options
} 

hyperparameter_tune_kwargs = { 
    'scheduler': 'local',
    'searcher': 'random'  
}

predictor_new_hpo = TabularPredictor(
    label="count", 
    verbosity=0,
    eval_metric='root_mean_squared_error'
).fit(
    train_data=train,
    time_limit=600, 
    presets="best_quality",
    hyperparameters=hyperparameters,
    hyperparameter_tune_kwargs=hyperparameter_tune_kwargs
)

predictor_new_hpo.fit_summary() 

*** Summary of fit() ***
Estimated performance of each model:
                    model  score_val  pred_time_val     fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  fit_order
0      CatBoost_BAG_L2/T1 -42.143245       0.140617     3.158673                0.006835           2.712248            2       True          6
1     WeightedEnsemble_L2 -43.369714       0.134118     0.521992                0.000336           0.075567            2       True          4
2     WeightedEnsemble_L3 -43.407207       0.437027  1016.738522                0.000264           0.075816            3       True          9
3      CatBoost_BAG_L2/T2 -43.751974       0.262332  1015.665507                0.128550        1015.219082            2       True          7
4  RandomForest_BAG_L1/T1 -44.340259       0.065923     0.276719                0.065923           0.276719            1       True          1
5    ExtraTrees_BAG_L2/T1 -44.938209       0.238166     0.802502                0.104384           0.356077            2       True          8
6  RandomForest_BAG_L2/T1 -45.322988       0.203829     1.087548                0.070047           0.641123            2       True          5
7      CatBoost_BAG_L1/T1 -45.454415       0.105544   505.513521                0.105544         505.513521            1       True          2
8    ExtraTrees_BAG_L1/T1 -45.547377       0.067859     0.169706                0.067859           0.169706            1       True          3
Number of models trained: 9
Types of models trained:
{'StackerEnsembleModel_XT', 'WeightedEnsembleModel', 'StackerEnsembleModel_RF', 'StackerEnsembleModel_CatBoost'}
Bagging used: True  (with 8 folds)
Multi-layer stack-ensembling used: True  (with 3 levels)
Feature Metadata (Processed):
(raw dtype, special dtypes):
('category', [])  : 6 | ['weather', 'month', 'hour', 'temp_category', 'humid_category', ...]
('int', [])       : 1 | ['day']
('int', ['bool']) : 3 | ['holiday', 'workingday', 'year']
Plot summary of models saved to file: AutogluonModels/ag-20220404_192217/SummaryOfModels.html
*** End of fit() summary ***

{'model_types': {'RandomForest_BAG_L1/T1': 'StackerEnsembleModel_RF',
  'CatBoost_BAG_L1/T1': 'StackerEnsembleModel_CatBoost',
  'ExtraTrees_BAG_L1/T1': 'StackerEnsembleModel_XT',
  'WeightedEnsemble_L2': 'WeightedEnsembleModel',
  'RandomForest_BAG_L2/T1': 'StackerEnsembleModel_RF',
  'CatBoost_BAG_L2/T1': 'StackerEnsembleModel_CatBoost',
  'CatBoost_BAG_L2/T2': 'StackerEnsembleModel_CatBoost',
  'ExtraTrees_BAG_L2/T1': 'StackerEnsembleModel_XT',
  'WeightedEnsemble_L3': 'WeightedEnsembleModel'},
 'model_performance': {'RandomForest_BAG_L1/T1': -44.34025936357441,
  'CatBoost_BAG_L1/T1': -45.454415170852016,
  'ExtraTrees_BAG_L1/T1': -45.54737689595722,
  'WeightedEnsemble_L2': -43.369714134052515,
  'RandomForest_BAG_L2/T1': -45.3229883681712,
  'CatBoost_BAG_L2/T1': -42.143245433872984,
  'CatBoost_BAG_L2/T2': -43.75197366935809,
  'ExtraTrees_BAG_L2/T1': -44.93820863029768,
  'WeightedEnsemble_L3': -43.4072066103833},
 'model_best': 'WeightedEnsemble_L2',
 'model_paths': {'RandomForest_BAG_L1/T1': 'AutogluonModels/ag-20220404_192217/models/RandomForest_BAG_L1/T1/',
  'CatBoost_BAG_L1/T1': 'AutogluonModels/ag-20220404_192217/models/CatBoost_BAG_L1/T1/',
  'ExtraTrees_BAG_L1/T1': 'AutogluonModels/ag-20220404_192217/models/ExtraTrees_BAG_L1/T1/',
  'WeightedEnsemble_L2': 'AutogluonModels/ag-20220404_192217/models/WeightedEnsemble_L2/',
  'RandomForest_BAG_L2/T1': 'AutogluonModels/ag-20220404_192217/models/RandomForest_BAG_L2/T1/',
  'CatBoost_BAG_L2/T1': 'AutogluonModels/ag-20220404_192217/models/CatBoost_BAG_L2/T1/',
  'CatBoost_BAG_L2/T2': 'AutogluonModels/ag-20220404_192217/models/CatBoost_BAG_L2/T2/',
  'ExtraTrees_BAG_L2/T1': 'AutogluonModels/ag-20220404_192217/models/ExtraTrees_BAG_L2/T1/',
  'WeightedEnsemble_L3': 'AutogluonModels/ag-20220404_192217/models/WeightedEnsemble_L3/'},
 'model_fit_times': {'RandomForest_BAG_L1/T1': 0.2767190933227539,
  'CatBoost_BAG_L1/T1': 505.5135214328766,
  'ExtraTrees_BAG_L1/T1': 0.16970586776733398,
  'WeightedEnsemble_L2': 0.07556676864624023,
  'RandomForest_BAG_L2/T1': 0.6411228179931641,
  'CatBoost_BAG_L2/T1': 2.7122480869293213,
  'CatBoost_BAG_L2/T2': 1015.2190816402435,
  'ExtraTrees_BAG_L2/T1': 0.356076717376709,
  'WeightedEnsemble_L3': 0.07581591606140137},
 'model_pred_times': {'RandomForest_BAG_L1/T1': 0.06592273712158203,
  'CatBoost_BAG_L1/T1': 0.10554361343383789,
  'ExtraTrees_BAG_L1/T1': 0.06785917282104492,
  'WeightedEnsemble_L2': 0.00033593177795410156,
  'RandomForest_BAG_L2/T1': 0.07004714012145996,
  'CatBoost_BAG_L2/T1': 0.006834983825683594,
  'CatBoost_BAG_L2/T2': 0.12855005264282227,
  'ExtraTrees_BAG_L2/T1': 0.10438394546508789,
  'WeightedEnsemble_L3': 0.0002639293670654297},
 'num_bag_folds': 8,
 'max_stack_level': 3,
 'model_hyperparams': {'RandomForest_BAG_L1/T1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'CatBoost_BAG_L1/T1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'ExtraTrees_BAG_L1/T1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'WeightedEnsemble_L2': {'use_orig_features': False,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'RandomForest_BAG_L2/T1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'CatBoost_BAG_L2/T1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'CatBoost_BAG_L2/T2': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True},
  'ExtraTrees_BAG_L2/T1': {'use_orig_features': True,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True,
   'use_child_oof': True},
  'WeightedEnsemble_L3': {'use_orig_features': False,
   'max_base_models': 25,
   'max_base_models_per_type': 5,
   'save_bag_folds': True}},
 'leaderboard':                     model  score_val  pred_time_val     fit_time  \
 0      CatBoost_BAG_L2/T1 -42.143245       0.140617     3.158673   
 1     WeightedEnsemble_L2 -43.369714       0.134118     0.521992   
 2     WeightedEnsemble_L3 -43.407207       0.437027  1016.738522   
 3      CatBoost_BAG_L2/T2 -43.751974       0.262332  1015.665507   
 4  RandomForest_BAG_L1/T1 -44.340259       0.065923     0.276719   
 5    ExtraTrees_BAG_L2/T1 -44.938209       0.238166     0.802502   
 6  RandomForest_BAG_L2/T1 -45.322988       0.203829     1.087548   
 7      CatBoost_BAG_L1/T1 -45.454415       0.105544   505.513521   
 8    ExtraTrees_BAG_L1/T1 -45.547377       0.067859     0.169706   
 
    pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \
 0                0.006835           2.712248            2       True   
 1                0.000336           0.075567            2       True   
 2                0.000264           0.075816            3       True   
 3                0.128550        1015.219082            2       True   
 4                0.065923           0.276719            1       True   
 5                0.104384           0.356077            2       True   
 6                0.070047           0.641123            2       True   
 7                0.105544         505.513521            1       True   
 8                0.067859           0.169706            1       True   
 
    fit_order  
 0          6  
 1          4  
 2          9  
 3          7  
 4          1  
 5          8  
 6          5  
 7          2  
 8          3  }

# Output the fit summary of the training run
predictor_new_hpo.leaderboard(silent=True).plot(kind="bar", x="model", y="score_val")

<AxesSubplot:xlabel='model'>

# Remember to set all negative values to zero
predictions = predictor_new_hpo.predict(test)
predictions = predictions.clip(lower=0)

# Same submitting predictions
submission_new_hpo = pd.read_csv('data/sampleSubmission.csv', parse_dates=['datetime'])
submission_new_hpo["count"] = predictions
submission_new_hpo['count'] = submission_new_hpo['count'].apply(lambda x: round(x))

submission_new_hpo.to_csv("submission_new_hpo.csv", index=False)
submission_new_hpo.head()

	datetime	count
0	2011-01-20 00:00:00	15
1	2011-01-20 01:00:00	7
2	2011-01-20 02:00:00	3
3	2011-01-20 03:00:00	2
4	2011-01-20 04:00:00	2

!kaggle competitions submit -c bike-sharing-demand -f submission_new_hpo.csv -m "new features with hyperparameters"

100%|████████████████████████████████████████| 149k/149k [00:02<00:00, 63.7kB/s]
Successfully submitted to Bike Sharing Demand

!kaggle competitions submissions -c bike-sharing-demand | tail -n +1 | head -n 6

fileName                     date                 description                        status    publicScore  privateScore  
---------------------------  -------------------  ---------------------------------  --------  -----------  ------------  
submission_new_hpo.csv       2022-04-04 20:10:27  new features with hyperparameters  complete  0.51299      0.51299       
submission_new_hpo.csv       2022-04-04 18:08:27  new features with hyperparameters  complete  0.50650      0.50650       
submission_new_features.csv  2022-04-04 17:33:48  new features                       complete  0.50239      0.50239       
submission.csv               2022-04-03 22:23:11  first raw submission               complete  1.80656      1.80656       

New Score of 0.50650

Step 7: Write a Report

Refer to the markdown file for the full report

Creating plots and table for report

# Taking the top model score from each training run and creating a line plot to show improvement
# You can create these in the notebook and save them to PNG or use some other tool (e.g. google sheets, excel)
fig = pd.DataFrame(
    {
        "model": ["initial", "add_features", "hpo"],
        "score": [-51.0252, -41.6628, -42.143245]
    }
).plot(x="model", y="score", figsize=(8, 6)).get_figure()
fig.savefig('model_train_score.png')

# Take the 3 kaggle scores and creating a line plot to show improvement
fig = pd.DataFrame(
    {
        "test_eval": ["initial", "add_features", "hpo"],
        "score": [1.80656, 0.50239, 0.50650]
    }
).plot(x="test_eval", y="score", figsize=(8, 6)).get_figure()
fig.savefig('model_test_score.png')

Hyperparameter table

# The 3 hyperparameters we tuned with the kaggle score as the result
pd.DataFrame({
    "model": ["initial", "add_features", "hpo"],
    "hpo1": ["default", "default", "CAT: {'iterations': 10000}"],
    "hpo2": ["default", "default", "RF: {'n_estimators': 300}"],
    "hpo3": ["default", "default", "XT: {'n_estimators': 300}"],
    "score": [1.80656, 0.50239, 0.50650]
})

	model	hpo1	hpo2	hpo3	score
0	initial	default	default	default	1.80656
1	add_features	default	default	default	0.50239
2	hpo	CAT: {'iterations': 10000}	RF: {'n_estimators': 300}	XT: {'n_estimators': 300}	0.50650